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University rankings historically have been based on a number of criteria, the most common being research output 
and quality. Different approaches have been developed which are surrounded by controversy; nevertheless, 
university rankings remain popular. It can be noted that due to the global economic recession, they receive further 
interest since it is believed that education is one of the sectors that countries should invest to improve economic 
development. The present paper presents a methodology for ranking universities based on research output and 
quality using as criterion by the CHC (Core of Human Capital) of universities demonstrating how the different 
positions of academic members of staff can contribute to the evaluation of their institutes at different time periods. 
The case study is the Greek Departments of Economics. Different scenarios are examined regarding the size of the 
department and time. The results offer conclusions in terms of university evaluation and strategic recruitment. In 
this respect, decision-makers can identify where they need to improve and if they are satisfied with their position on 
the education map. 
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Introduction 

The use of rankings in HE (higher education) is an attractive but also controversial subject. Although the 
history of rankings is not long, it began in 1983 in the US, they have become an important feature in the 
decision- and policy- making in HE. They serve different purposes and address different audiences. As Benito 
and Romera (2011) explained, rankings serve to produce a discussion on what quality in HE means influencing 
the perceptions on university excellence. In this respect, rankings complement the work undertaken by public 
and independent accrediting agencies in the context of quality assessment and review. Although rankings may 
not be universally accepted, it is increasingly acknowledged that they are an important tool for students, parents, 
academics, potential employers, and the public in general. 

As such, nowadays, it is almost impossible to discuss about the evaluation of institutions and compare 
them without the use of rankings, particularly when the scientific community itself acknowledges that its output 
must be quantified to allow for comparisons and fair allocation of resources and funding. Many authors (for 
example, Bernardino & Marques, 2009) pointed out that, although institutions do not generally like to be 
ranked or compared with others, and despite the criticism that academic rankings receive, the mission for 
several of them is to be at the top of an academic ranking table. This is because top-class rankings offer good 
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reputation and free publicity to the institutions and therefore attract the best students. They also assist funders in 
their decisions as to which university provides the best “investment”. For example, Boeing announced that it 
will use university perfonnance data to determine which institutes will share US$100 million available for 
training and other activities, and in the same respect, Deutsche Telecom has admitted that its decision on 
professional chairs is influenced by rankings (Hazelkom, 2009). Furthermore, the success of departments to 
rankings is often employed as decision-making criteria by university marketing divisions to allocate education 
funds (Kalaitzidakis, Mamuneas, Theofanis, & Stengos, 2003). 

The use of rankings has led to many discussions about the benefits and costs of their use. Academic 
rankings are frequently criticized on the indicators employed, weighting and normalization methods applied 
(Boulton, 2011; Benito & Romera, 2011; Harvey, 2008). There are also several questions about the quality, 
impact and actual “value” of the outputs and thus on the conclusions produced. Considering that universities 
are complex organizations with strengths and weaknesses that vary across their different departments, it is clear 
that excellence in each case is based on different criteria and indicators which causes concern on the 
consistency of results and nevertheless justifies the diversity of ranking approaches. 

In recognition of the these concerns on rankings, the UNESCO-CEPES (United Nations Educational 
Scientific and Cultural Organisation European Centre for Higher Education) and the Institute for Higher 
Education Policy initiated in 2004, the IREG (International Ranking Expert Group) which developed a set of 
principles of quality and good practice in Higher Education Institute rankings — the Berlin Principles on 
Ranking of Higher Education Institutions (CHE (Centre for Higher Education Development), 
UNESCO-CEPES, & IHEP (Institute for Higher Education Policy), 2006). Sixteen descriptive and prescriptive 
principles for ranking are included in the principles and cover four aspects which are: the purpose and goal of 
ranking; the design and weighting of indicators; the collection and processing of data; and the presentation of 
ranking results. The principles aimed to shift the interest away from the ranking problems. The focus was 
placed on the benefits that can drive from rankings and included a set of recommendations about what should 
be done when an organization produces a ranking which provides useful conclusions to all interested parties 
(Benito & Romera, 2011). 

National rankings have been created in over 40 countries (Hazelkorn, 2009). International rankings are 
less in number and more recent but they have become more influential. The most famous and respected HE 
rankings are the THES (Times Higher Education) which began in 2004 and the SJTU (Shanghai ranking) 
commenced in 2003. The two rankings differ largely in their evaluation method. The THES ranking (which 
recently started co-ordination with Thomson Reuters) places significant importance on subjective evaluations 
by experts and recruiters, directed towards the prestige and power of the university. More analytically, there are 
five basic categories under which most criteria are classified namely teaching, research, citations, industry 
income, and international outlook. This approach is heavily criticised by Marginson (2007) who supported that 
the THES ranking does not examine the research dimension in depth and as a result produces tables that do not 
express the quality of education. Conversely, the SJTU does not have any indicator measured by surveys and 
opinions. All the Shanghai indicators are quantitative measures rather than qualitative ones. 

Europe, in response to the legitimate concerns on rankings and their weaknesses, has initiated two projects: 
The U-Map which aims to map the diversity of institutions and the U-Multirank which aims to produce 
international ranking of university perfonnance in dimensions similar to those used to map diversity. Although 
the value of these approaches is recognized, they are treated with skepticism recognizing the difficulty in 
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collecting the large volumes of pertinent data and the use of inter-country statistics (Boulton, 2011). 

Very often, the indicators of most academic rankings are based on what can be measured, such as 
publications, citations, and indicators that are less tangible, such as educational quality, are neglected (Meredith, 
2008). As such, the research performance of academics has always been an important indicator for the ranking 
evaluation since it is relatively easier to measure, for example, the number of articles published in international 
peer review journals, than less tangible indicators, such as the teaching quality. By focusing however primarily 
on research intensity, the rest of dimensions, i.e., teaching, innovation, staff/student ratios, and social and 
economic impact are overlooked. As Butler (2010) indicated, the recent trend to university rankings approaches 
shifted away from basic assessment methodologies and towards more subtle assessments using more criteria 
(even purely qualitative factors, such as educational excellence) providing an overall better guidance to all 
interested parties. He further argued that it is rather narrow to evaluate institutes only on their research 
performance because in this case institutes are isolated from the totality of benefits that offer to society. As the 
LERU (League of European Research Universities) strongly claims, research cannot be the only benefit that 
universities offer to society and thus evaluation methods should extend their criteria to factors other than 
research. Frey and Rost (2010), for example, admitted that rankings based on publications and citations do not 
necessarily effectively reflect research quality, and proposed an alternative ranking based on membership on 
academic editorial boards of professional journals. 

Nevertheless, at present, in the US, the basic evaluation criterion for a member of staff is the research 
productivity based on the number and quality of publications (impact factor, citations) and the evaluation takes 
place internally. In Europe, and in particular, in the UK and the Netherlands, the research evaluation is 
undertaken by scientific committees of recognizable scientific work on the subject and besides research, other 
indicators such as contribution to society and other positions are considered. A comparative study between the 
US and European ranking systems is presented by Frey and Eichenberger (1993). Whether the evaluation of 
departments should take place in house or be outsourced is another issue of controversy which is not within the 
scope of the present discussion (for more infonnation on the subject see Enserink, 2007). In the Greek 
academic environment, it is only recent the legislation according to which universities will be evaluated by a 
third party; before this, there was no consistent way of ranking universities, at least dictated by a central 
governmental body. It is yet to be seen how this legislation will come into force and how the actual application 
will take place. 

Several types of university rankings exist, as will be discussed in the next section, the majority of which 
examine only the research indicator (for an extensive discussion see Boulton, 2011). A methodology which is 
widely accepted and satisfies public interest, students, and academics has not yet been developed. The present 
research acknowledges that research indicators alone are too narrow to be used for the ranking of universities. 
For this purpose, the researchers have supported a unified system of assessment which takes into consideration 
both research and education quality (Giannias & Sfakianaki, 2011). However, the present paper exploits further 
the research dimension which can be integrated in the proposed unified system at a later stage. In particular, it 
examines an indicator which is very little discussed in the literature and is related to the importance of the 
research performance of different academic positions, i.e., professors, assistant professors, etc., within an 
institution at different time periods. In this respect, the methodology originally developed by Giannias and 
Sfakianaki (in press) will be extended and conclusions and guidance will be brought in for an approach to a 
Greek ranking since the research will be applied to the Greek Departments of Economics. The research 



SHORT AND LONG RUN UNIVERSITY RANKINGS 


729 


provides contributions for the literature since ranking issues using different academic positions and their 
research perfonnance in due time, is hardly exploited. The ranking is developed on the basis of results derived 
from the EconLit and SSCI (Social Science Citation Index) for publications over the period of 1969-2004 and 
the citations they received since their publication respectively. 

University Rankings 

The strong interest in HE rankings by academics, employers, students, and society in general, as a need to 
evaluate academic output, has been demonstrated in the previous section. The basis of most methods has been 
founded on measures such as citations and impact factors. A detailed presentation of most of the important 
methods and the indicators employed is presented by Giannias and Sfakianaki (2011). In this respect, in the 
present paper the pertinent discussion is short. 

Ranking methods based on research evaluation use several criteria, the most important of which is the 
journal in which the publication is included, the number of publications and the number of citations. For 
example, Kalaitzidakis, Mamuneas, and Stengos ( 1 999) in an influential paper, used publications in a core set 
of 10 economic journals from 1991 to 1996 to rank European economics institutions and countries. Their 
research demonstrated the three leading universities in Europe and the three top-ranked countries. Coupe (2002) 
in his report employed performance indicators available in the literature (e.g., weighted journal ranking) to 
assess the work of individual researchers concluding on the ranking of economics departments on an 
international basis. The database of the research covers the period of 1 969-2000 for economists and the period 
of 1990-2000 for institutions. Scott and Mitias (1996) and Dusansky and Vernon (1998) presented US 
Economics Department rankings whereas Jin and Yau (1999) and Jin and Hong (2008) presented rankings of 
Asian departments. Canadian rankings are discussed by Lucas (1995) and Davies, Kocher, and Sutter (2008); 
Australian by Towe and Wright (1995) and Neri and Rodgers (2006); European by Lubrano, Bauwens, Kirman, 
and Protopopescu (2003). 

Jin and Hong (2008), in their research on ranking economic research in East Asia, used 60 journals over 
the period of 1990-2005 using EconLit for their database. The selection of the 60 journals was based on a 
number of sources, such as the selection proposed by Dusansky and Vernon (1998), the selection proposed by 
Scott and Mitias (1996), and three best regional journals in Asia. Kalaitzidakis et al. (1999), in their research 
focused on 10 journals, such as AER (American Economic Review), Econometrica, Journal of Political 
Economy, etc., tried to overcome the issue of journal quality, for the period of 1991-1996. The uncertainty that 
is inherent in the methods described above is the period of examination. More specifically, the proposed 
criterion, where a pre-defmed list of journals is used, is convenient if the period of examination is short. On the 
contrary, if the period is long, then the validity of this indicator is disputed since it is difficult to evaluate the 
quality of a journal in the long-tenn. In this case, simple recording of publications listed in the EconLit is 
sufficient irrespective of the journal of publication and its impact factor. 

The number of publications and/or the pages per publication is a factor that is of significance in evaluation 
methods. The challenge in this case is how the pages are counted since journals have different page sizes and 
naturally, quality. One way to overcome this problem is the conversion of all journal pages to equivalent units 
using the AER size of page as a standard to unify size. Kalaitzidakis et al. (1999), for example, employed the 
Laband-Piette approach and believed that the counting of pages per article using this conversion method 
reflects better the quantity of the research work produced and the overall effort (Laband & Piette, 1994). 
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Dusansky and Vernon (1998) also applied the Laband-Piette approach in their publication-based approach 
where the top 50 US economic departments are presented. Jin and Hong (2008), for the articles published in the 
60 journals selected, counted the total number of pages and converted the total pages published to the 
equivalent length of page of the AER. In the same respect, Jin (2005) also counted the pages of articles 
published, during his ranking having ensured however that articles employed are being referred. 

The number of citations is also frequently used when evaluation is based on research. Kalaitzidakis et al. 

( 1 999) suggested, as an alternative way to measure the quality of articles, the use of the actual citation impact 
of each article. Although the authors have expressed their preference to the number of pages per article, they 
argued that citations could indicate institutions which produce relatively few articles which are highly cited 
compared with institutions which produce a lot of articles that do not stay long in the citations index. Further, 
Coupe (2002) proposed the use of impact factors (available for 273 journals) that derive after counting the 
number of citations for an article published in a journal. The measurement takes place one to two years after 
publication. A difficulty with this measure is that citations are weighted equally; the contribution of a citation 
from a top journal is considered the same as a citation from a lower ranked journal. Considering that citations 
require time to build up, this criterion is an asset for the old articles compared to recent ones. The present 
research supports the use of simple count of citations of individual articles and avoids any weighting of journals. 
In this respect, a measure of the quality of the article is provided (while the impact factors provide a measure of 
the quality of the journal where an article is published). 

Methods which use other indicators besides research, such as educational quality, staff/student rations, 
exam entry scores, have also been developed to a lesser extend however (see Giannias & Sfakianaki, 2011; 
Lukman, Krajnc, & Glavic, 2010; Jin, 2005). Bitros and Karayiannis (2000) developed a well-structured 
framework aimed to the Greek academic environment where they take into consideration not only the research 
productivity of the member of staff, but also the teaching quality, the administration work undertaken, and the 
social contribution. The methodology although well documented, requires the collection of different sources of 
data, in some cases non-tangible. Although, as Butler (2010) supported, the trend of ranking universities moves 
towards the direction where research is not the only indicator to examine in developing rankings, in countries, 
such as Greece where there is no tradition in ranking universities and there is no officially established system, 
the collection of data proposed by Bitros and Karayiannis (2000), can be a difficult and also controversial issue. 

Irrespective of the indicators employed, in all cases of evaluation, departments are being evaluated on the 
perfonuance of the total number of academic members irrespective of their position. It seems that there is not 
yet a methodology that examines the contribution of each position of academic members of staff in different 
time periods as will be further defined in the next section. 

A Conceptual Model for Ranking Universities 

Giannias and Sfakianaki (in press) presented a conceptual model for the ranking of the Departments of 
Economics in Greece. The evaluation of departments was based on the research output and quality records' of 
all academic members of each department who have tenure or are on a tenure track. In the present study, the 
basic research hypothesis is still the same, i.e., the research employs research output and quality, however, in 
the current study, the assessment of the Greek departments is implemented separately for each period of 

1 The research output was quantified by the per capita number of articles published in EconLit listed articles and citations shown 
in SSCI for publications over the period of 1969-2004. 
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academic staff for four periods of time. The hypothesis is therefore extended to demonstrate the short- and 
long- term impact of each position in the overall assessment of the department. The indicators employed for the 
development of the research is the number of EconLitR (publications) irrespective of the journal of publication, 
its impact factor, and the number of pages; and the number of C (citations) per publication without assigning 
any impact factors. 

For each Department of Economics in Greece and for each position of academic staff, PA, we compute its 
mean research output P(PA) and its mean quality of EconLit publications C(PA), where: 


P(PA) 


2 >(^4) 
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n 
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(2) 


n 

i = 1, .... n refers to the lull-time academic members of staff of position PA of a department, PA refers to 
the academic position; PA = PR (professor), AP (associate professor), AsP (assistant professor), and L 
(lecturer) , p,( PA ) is the number of publications of faculty member i of position PA, Ci(PA) is the number of 
citations that the publications of author i of position PA have received. 

All values are mapped in to a 0-100 scale; where, 100 is mapped to the best value of a variable, and 0 to the 
worse as shown in Equation (3): 

x ._ (X-Xmin) , m 

(Xmax - Xmin) u ) 


where, X= P(PA), C{PA). 

To accommodate our comparisons, we introduce a composite index, CI(PA), for each position, PA, which 
combines the weighted mean of P(PA) and C(PA) values for each department using the w p and w c weights 
respectively. These are obtained from an expert’s opinion survey , and are the following: 


w p = 0.60 and w c = 0.40 

Subsequently, the following Composite Index, CI*(PA), is computed for each Department of Economics, 
where: 

CI*(PA ) = w p P *( PA ) + w c C*(PA) (4) 

In terms of t (time), four periods are considered dividing time in current and long-term periods. More 
specifically, first we examine the SR (short run) period; we may think of it as being the current situation 
(“today”). LR1 period (Long Run Period One) follows which is defined as the period in which the professors of 
period SR have been retired and the remaining academic staff holding a position in period SR have been 
advanced to the next rank; that is, an associate professor in period SR has been promoted to professor in period 
LR1, an assistant professor in period SR has been promoted to associate professor in period LR1, and a lecturer 


2 The four positions of academic staff in Greek universities, by state law, are: PR (professor), associate professor ( AP), AsP 
(assistant professor), and L (lecturer). Professors and associate professors have tenure whereas assistant professors and lecturers 
are on a tenure track. It is important to mention that with the recent law change in the educational sector (September 2011), the 
academic positions that are on a tenure track have decreased from two to one. 

3 Thirty Greek experts participated in a survey undertaken by the authors, each of which: (1) had a minimum of 10-year research 
experience in social science research; and (2) was a member of a committee for the evaluation of applicant for teaching positions in 
more than five occasions. 
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in period SR has been promoted to an assistant professor in period LR1. LR2 period (Long Run Period Two) is 
the next period: It is defined as the period in which the professors and the associate professors of period SR 
have been retired and the remaining academic staff holding a position in period SR have been advanced to the 
second higher rank; that is, an assistant professor in period SR has been promoted to professor in period LR2, 
and a lecturer in period SR has been promoted to an associate professor in period LR2. LR3 period (Long Run 
Period Three) is the final period: It is defined as the period in which the professors, the associate professors, 
and the assistant professors of period SR have been retired and the remaining academic staff holding a position 
in period SR have been advanced to the third higher rank; that is, a lecturer in period SR has been promoted to 
professor in period LR3. 

Given Equation (4) and taking the weighted mean of the CI*(PA) for a Department of Economics, we 
obtain its OCI* (Overall Performance Composite Index), according to our criteria as illustrated in Equation (5): 

OCI* = £ X(PA) x Cl * (PA) (5) 

PA 

where X( PA ) is the number of academic staff of position PA in period t, as a percentage of the total number of 
academic staff of the same period t where t = SR, LR1, LR2, LR3. 

Clearly, Equation (5) can be employed to compute the OCI * score of a department at any time period; 
namely, OCI*(SR), OCI*(LRl), OCI*(LR2), and OCI*(LR3) for periods of SR, LR1, LR2, and LR3, 
respectively. The applicability of the conceptual model was tested for the nine Departments of Economics in 
Greece. These are the Departments of Economics of: AUEB (Athens University of Economics and Business), 
UCRE (University of Crete), CAPO (National and Capodistrian University of Athens), UMAC (University of 
Macedonia), UTHE (University of Thessaly), UP AT (University of Patras), UPIR (University of Piraeus), 
UIOA (University of Ioannina), and ARIS (Aristotle University of Thessaloniki). 

In the next sections, the performance of each department in periods of SR, LR1, LR2, and LR3 is evaluated. 
For these evaluations, the following assumptions have been made; the research output productivity (in tenns of 
both quantity and quality) of each academic staff remains constant through time, academic staff do not move 
but only retire, and the number of members of academic staff that a department has at period SR is taken as a 
basis for the application of Equation (5) at all time periods. The evaluation of the nine Greek departments, 
which is presented in the following section, is based on their research output and quality using 2004 EconLit 
and SSCI data. 


Results 

Short and Long Run Rankings Based on an Evaluation of the CHC (Core of Human Capital) of Each 
Greek Department of Economies 

In the following, we evaluate departments at various periods (present and future) using the Composite 
Index, Equation (5), considering only those academic members who belonged to the department in period SR. 
To do this we have assumed that the CHC (Core of the Human Capital) of a department in subsequent periods 
consists of members who were in the department in period SR 4 . The basic assumptions of the present analysis 
are the following: 

(1) The CHC in period SR, CHC(SR), consists of the PR, AP, AsP, and L of period SR; 


4 The quality of the CHC of a period (as it is evaluated herein) determines the recruiting needs for new academic staff. 
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(2) The CHC in period LR1, CHC(LR1), consists of the PR, AP, AsP, of period LR1 who were 
(presumably) AP, AsP, and L in period SR; 

(3) The CHC in period LR2, CHC(LR2), consists of the PR, AP of period LR2 who were (presumably) AP, 
AsP in period LR1 and AsP, and L in period SR; 

(4) The CHC in period LR3, CHC(LR3), consists of the PR of period LR3 who were (presumably) AP in 
period LR2, AsP in period LR1, and L in period SR. 

For example, the OCI* value for the UTHE is based on the structure of its CHC in each period, which is 
illustrated in Table 1; the figures in parenthesis give the members of staff per academic position as a percentage 
of the total. 

Table 1 

Structure of CHC per Period and Academic Position for the UTHE, Department of Economics — Number of 
Academic Members Changes With Time 


UTHE, Department of Economics 
Number of members of staff per PA and t 


Period SR 





PR 

AP 

AsP 

L 

Total 

3 (0.33) 

1(0.11) 

4 (0.44) 

1(0.11) 

9(1.00) 

Period LR1 

PR 

AP 

AsP 

Total 


1 (0.17) 

4 (0.67) 

1 (0.17) 

6(1.00) 

Period LR2 


PR 

AP 

Total 



4 (0.80) 

1 (0.20) 

5(1.00) 

Period LR3 



PR 

Total 




1 (1.00) 

1 (1.00) 


Table 2 presents the OCI* values for the CHC of the nine DEPT (Departments of Economics of Greek 
Universities) for the four periods of time under consideration. These values were obtained after applying 
Equations ( 1) — (5). For each period, universities are ranked (R) from higher to lower values and a comparison 
between the SR and LR3 was undertaken; in the case that the department falls into a lower position then it is 
marked grey whereas in the case that it moves to a higher position it is marked with horizontal hatch. Finally, 
when the rank remains the same, there is no color marking. Ceteris paribus, the higher the position of a 
department, the higher is the contribution of today’s members of staff (period SR) in the future position of the 
department in the relevant rankings (periods LR1, LR2, and LR3). Ceteris paribus, the lower the position of a 
department on the ranking is, the greater the needs to attract the relative best candidates for the open positions 
through recruiting. 

In an attempt to quantify the difference between the “best” department and the rest, the percentage 
difference between the first and each of the rest is evaluated for each period under examination. In each case, 
universities are ranked using their OCI* values from best to worst. The results are presented in Table 3 for the 
periods of SR, LR1, LR2, and LR3 respectively. In the first case (SR), it is clear that the gap between the first 
(AUEB) and the last (ARIS) is large and reaches approximately 94%. The universities, based on their 
difference from the best one, can be divided into four groups: difference 37%~48% for the UCRE, CAPO, and 
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UPAT, difference from 56%-61% for the UMAC and UIOA, difference 75%-79% for the UTHE and UPIR, 
and more than 90% for the ARIS. The standard deviation of OCI* in period SR is 24.52 and indicates well 
spread values around the mean which in turn illustrates the differences in perfonnance between the various 
departments. 


Table 2 

OCI* Values for CHC for the Four Periods (SR, LR1, LR2, and LR3) and Comparison of Ranking Between SR 
and LR3 


DEPT 


AUEB 

UCRE 

CAPO 

UPAT 

UMAC 

UIOA 

UTHE 

UPIR 

ARIS 



Table 3 


Percentage Difference (A%) Between the Best and the Rest Based on the OCI* Values for the Departments of 
Economics of Greek Universities for Periods SR, LR1, LR2, and LR3 


R 


SR 



LR1 



LR2 



LR3 


DEPT 

OCI* 

A% 

DEPT 

OCI* 

A% 

DEPT 

OCI* 

A% 

DEPT 

OCI* 

A% 

1 

AUEB 

89 


AUEB 

12 


AUEB 

70 


UPAT 

19 


2 

UCRE 

55 

37.91 

UPAT 

56 

21.54 

UPAT 

59 

16.17 

AUEB 

76 

4.16 

3 

CAPO 

53 

40.48 

CAPO 

49 

31.72 

UMAC 

48 

31.38 

UIOA 

57 

28.64 

4 

UPAT 

47 

47.25 

UCRE 

45 

37.8 

CAPO 

42 

40.37 

CAPO 

43 

45.93 

5 

UMAC 

39 

56.36 

UMAC 

39 

45.91 

UIOA 

38 

45.14 

UMAC 

26 

67.06 

6 

UIOA 

36 

60.03 

UIOA 

36 

50.42 

UCRE 

36 

48.84 

UCRE 

17 

78.04 

7 

UTHE 

22 

75.86 

UPIR 

29 

59.82 

UPIR 

24 

65.98 

ARIS 

11 

86.47 

8 

UPIR 

19 

78.2 

UTHE 

16 

78.01 

UTHE 

7 

90.37 

UTHE 

0 

100 

9 

ARIS 

6 

93.21 

ARIS 

4 

93.78 

ARIS 

5 

93.32 

UPIR 

0 

100 


In the case of LR1, the gap between the first (AUEB) and the last institutes (ARIS) remains large and 
reaches again approximately 94%. For LR1, the universities, based on their difference from the best one, can be 
divided into the following groups: difference 21% — 23% for the UPAT, difference 3 1 % — 3 8% for the UCRE and 
CAPO, difference 45%-60% for the UMAC, UIOA, and UPIR, difference 78%-79% for the UTHE, and more 
than 90% for the ARIS. In this case, the standard deviation is 20.50 which is lower than in the case of SR, but 
only slightly and thus the conclusions remain the same. 

In the case of LR2, the divisions of groups is as follows: difference 1 6% — 18% for the UPAT, difference 
3 1 %— 33% for the UMAC, difference 40%-49% for the CAPO, UIOA, and UCRE, difference 65%-67% for 
the UPIR, and more than 90% for UTHE and the ARIS. In this case, the distance between the first and the last 
university remains large and concerns the same institutes, the first two universities are the same as in cases of 
SR and LR1 and the standard deviation is similar to SR and LR1, approximately 22. 
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In the case of LR3, the first and the last institutes are now different. More specifically, the first is the 
UPAT with AUEB being very close, whereas the last are the UTHE and UPIR. The rest of the groups are 
formed as follows: difference 4%-6% for CAPO, difference 28%-30% for the UIOA, difference 45%-47% for 
the CAPO, difference 67%-87% for the UMAC, UCRE, and ARIS, and 100% for UTHE and UPIR. In this 
case, the standard deviation is 3 1 indicating a greater spread of values around the mean than in the cases of SR, 
LR1, and LR2\ which shows that in the long-period LR3 the differences are enlarged. 

Short and Long Run Rankings Based on an Evaluation of the CHC of Each Greek Department of 
Economics Within a Department of Fixed Size 

The second application examines only those academic members who were members of a department in 
period SR, assuming however in this case that the size of the department does not change through time. OCI* 
values are calculated using Equation (5) and an example of the structure of CHC per period and academic 
position is demonstrated in Table 4 for the UTHE; the figures in parenthesis give the members of staff per 
academic position as a percentage of the total. 

Table 4 

Structure of CHC per Period and Academic Position for the UTHE, Department of Economics — Number of 
Academic Members Does Not Change With Time 


UTHE, Department of Economics 
Number of members of staff per PA and t 


Period SR 





PR 

AP 

AsP 

L 

Total 

3 (0.33) 

1(0.11) 

4 (0.44) 

1(0.11) 

9(1.00) 

Period LR1 

PR 

AP 

AsP 

Total 


1(0.11) 

4 (0.44) 

1(0.11) 

9 (0.67) 

Period LR2 


PR 

AP 

Total 



4 (0.44) 

1(0.11) 

9 (0.56) 

Period LR3 



PR 

Total 




1(0.11) 

9(0.11) 


The results in Table 5 show the contribution of the CHC to the OCI* of each department for a given size 
(that of period SR). For each period, universities are ranked from higher to lower values and a comparison 
between the SR and LR3 is undertaken; in the case that the department falls into a lower position, then it is 
marked grey whereas in the case that it moves to a higher position, it is marked with horizontal hatch. Finally, 
when the rank remains the same, there is no color marking. Assuming the size of the department remains the 
same as in period SR, the lower the position of a department on the ranking, the greater the needs to attract the 
relative best candidates for the open positions through recruiting. 

Universities are ranked using their OCI* values from best to worst and the difference between the “best” 
department and the rest are calculated using the percentage difference between the first and each of the rest for 
each period t. The results are presented in Table 6 for all periods. For period SR, the results remain the same, as 
presented in Table 3, since the number of members of staff for this time period has not changed in the 
assumptions of this case. 
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Table 5 

OCI* Values for CHC for the Four Periods (SR, LR1, LR2, and LR3 ) and Comparison of Ranking Between SR 
and LR3 


DEPT 


UTHE 

UPIR 

ARIS 


SR 


OCI* 


22 

19 

6 


LR1 


LR2 


OCI* 


OCI* 


3 

45 

36 



OCI* 



LR3 


AEUB 

89 

1 

28 

5 

24 

3 

15 

3 

UCRE 

55 

2 

29 

4 

17 

4 

1 

7 

CAPO 

53 

3 

33 

3 

16 

5 

8 

4 

UPAT 

47 

4 

20 

6 

4 

7 

1 

6 

UMAC 

39 

5 

11 

8 

4 

8 

0 

8 

UIOA 

36 

6 

17 

7 

9 

6 

0 

8 


It is interesting to notice that in the case that the number of academic members does not change through 
time, the results are quite different in all cases. For example, although the gap between the first and the last 
remains the same (approximately more than 90%), the institutes that are now best and worst are very different 
from the ones examined in section “Short and Long Run Rankings Based on an Evaluation of the CHC of Each 
Greek Department of Economics”. More specifically, in the cases of LR1 and LR2, the best institute is UPIR 
and the worst is UTHE. In the case of LR3, the best institute is ARIS and the worst is UIOA. It is worth 
mentioning that UPIR and ARIS are ranked towards the end of the list in the application of section “Short and 
Long Run Rankings Based on an Evaluation of the CHC of Each Greek Department of Economics”. In the 
analysis of this section, where the number of member of staff remains unchanged through time, the groups are 
formed in the following way. For LR1, difference 2 1 % — 27% for the ARIS and CAPO, difference 35%— 38% for 
the UCRE and AUEB, difference 55%-64% for the UPAT and UIOA, difference 76%-78% for the UMAC, 
and more than 90% for the UTHE. In this case, the SD (standard deviation) is 13.12 which is much lower than 
in the case of SR, showing that the values around the mean are less dispersed and thus there is convergence on 
the performance of the various departments. 

Table 6 


Percentage Difference (A%) Between the Best and the Rest Based on the OCI * Values for the Departments of 
Economics of Greek Universities for Periods SR, LR1, LR2, and LR3 


D 


SR 



LR1 



LR2 



LR3 


K 

DEPT 

OCI* 

A% 

DEPT 

OCI* 

A% 

DEPT 

OCI* 

A% 

DEPT 

OCI* 

A% 

1 

AUEB 

89 


UPIR 

45 


UPIR 

35 


ARIS 

33 


2 

UCRE 

55 

37.91 

ARIS 

36 

21.01 

ARIS 

35 

0.02 

UPIR 

32 

3.91 

3 

CAPO 

53 

40.48 

CAPO 

33 

26.18 

AUEB 

24 

31.18 

AUEB 

15 

55.72 

4 

UPAT 

47 

47.25 

UCRE 

29 

35.87 

UCRE 

17 

52.14 

CAPO 

8 

76.8 

5 

UMAC 

39 

56.36 

AUEB 

28 

38.72 

CAPO 

16 

55.54 

UTHE 

3 

89.51 

6 

UIOA 

36 

60.03 

UPAT 

20 

55.04 

UIOA 

9 

74.23 

UPAT 

1 

96.56 

7 

UTHE 

22 

75.86 

UIOA 

17 

63.42 

UPAT 

4 

88.14 

UCRE 

1 

96.9 

8 

UPIR 

19 

78.2 

UMAC 

11 

76.64 

UMAC 

4 

89.36 

UMAC 

0 

100 

9 

ARIS 

6 

93.21 

UTHE 

3 

92.33 

UTHE 

3 

90.15 

UIOA 

0 

100 
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In the case of LR2, the divisions of groups are as follows: difference less than 1% for the ARIS, difference 
3 1 %— 3 3 % for the AUEB, difference 52%-75% for the UCRE, CAPO, and UIOA, difference more than 88% 
for the UP AT, UMAC, and UTHE. In this case, the standard deviation is similar to LR1, approximately 12.73 
and enhances the conclusion of the convergence on the performance as in the case of LR1. 

In the case of LR3, the groups are, difference 4%-6% for the UPIR, difference 55%-77% for the AUEB 
and CAPO, difference more than 89% for UTHE, UP AT, UCRE, UMAC, and UIOA. In this case, the SD is 
13.34, which is a similar to LR1 and LR2. Table 6 indicates again that there are many universities with low 
scores and thus greater difference from the top university, which is more evident for periods further in time we 
examine ( LR3 ). However, in this application, the distance from the top university has been reduced in some 
occasions (for example, ARIS in LR2 ). 

When the results from Tables 3 and 6 are being compared, it is concluded that the results are quite 
different and it is worth emphasizing that although UPIR and ARIS are very low ranked in Table 3, their 
position in Table 6 has been considerably improved. On the contrary, AUEB is not the first or even close to the 
first universities whereas UTHE remains low ranked in both tables. The impact of the basic assumption 
whether the number of members of staff changes or not through time influences significantly the ranking of 
most universities. 

The results of Tables 3 and 6 should be investigated jointly in order to reach conclusions for a specific 
institution. For example, a high position in Table 3 and low position in Table 6 may be the result of high quality 
academic staff, the majority of which was retired at the end of period SR (so that the remaining are still good 
but very little relevant to the size of the department). On the contrary, a low position in Table 3 and high 
position in Table 6 may be the result of low quality academic staff, the majority of which was retired at the end 
of period SR, while the remaining are of pretty good quality and quite a few relevant to the size of the 
department. 

Recruiting Needs Based on OCI* Figures 

This section investigates the relative difficulty with which a department can retain its OCI* performance 
of present (SR period) in the future (periods of LR1-LR3). Based on the results of Table 5, the following 
differences are calculated: 

A LR 1 = OCI * ( LR 1) - OCI * (SR ) 

MR 2 = OCI * (LR 2) - OCI * (SR ) 

ALR3 = OCI * (LR3) - OCI * (SR) 

which are illustrated in Table 7. The greater (lower) the ALRi value for a department where i = 1, 2, 3, the 
greater (lower) its ability to achieve its objectives in period i, namely, to retain its OCI* performance of present 
(period SR) in period LRi. Furthermore, the lower the ALRi value of a department, the greater the need to attract 
the best teaching and research staff for reaching its objective (holding its OCI* performance to the OCI* level 
of period SR). The results are demonstrated in Table 7. 

Once, the results of Table 7 are shorted as illustrated in Table 8, the following conclusions derive: (1) The 
position of the departments in the four higher positions of the above ranking (ARIS, UPIR, UTHE, and UIOA) 
remains stable. The same holds for AUEB that holds the lower position in the above ranking in all periods; (2) 
From periods LRI to LR3, UMAC improves its position in the above ranking; (3) From periods LRI to LR3, 
UCRE’s position in the above ranking is lowered; and (4) From periods LRI to LR3, the position on the 
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relevant ranking of UP AT remains unchanged; however, its position in the above ranking is lower temporarily 
in LR2. 

Table 7 


Difficulty to Retain OCI* Performance in the Future 


DEPT 

ALR1 

R(ALR1) 

ALR2 

R(ALR2) 

ALR3 

R(ALR3) 

AUEB 

-62 

9 

-65 

9 

-75 

9 

UCRE 

-26 

6 

-39 

7 

-54 

8 

CAPO 

-20 

5 

-37 

6 

-45 

6 

UPAT 

-21 

7 

-43 

8 

-46 

7 

UMAC 

-28 

8 

-35 

5 

-39 

5 

UIOA 

-19 

4 

-27 

4 

-36 

4 

UTHE 

-18 

3 

-18 

3 

-18 

3 

UPIR 

26 

2 

16 

2 

12 

2 

ARIS 

30 

1 

29 

1 

27 

1 

Note. R: rank. 







Table 8 

Conclusions 

on the Positioning of Greek Departments of Economics — Future Cases 


Rank 

DEPT 

ALR1 

DEPT 

ALR2 

DEPT 

ALR3 

1 

ARIS 

30 

ARIS 

29 

ARIS 

21 

2 

UPIR 

26 

UPIR 

16 

UPIR 

12 

3 

UTHE 

-18 

UTHE 

-18 

UTHE 

-18 

4 

UIOA 

-19 

UIOA 

-27 

UIOA 

-36 

5 

CAPO 

-20 

UMAC 

-35 

UMAC 

-39 

6 

UCRE 

-26 

CAPO 

-37 

CAPO 

-45 

7 

UPAT 

-27 

UCRE 

-39 

UPAT 

-46 

8 

UMAC 

-28 

UPAT 

-43 

UCRE 

-54 

9 

AUEB 

-62 

AUEB 

-65 

AUEB 

-75 


Assuming the size of the department, as well as its OCI* score, remains that of period SR, the lower the 
position of a department on the ranking, the greater its needs to attract the relative best candidates for its open 
positions through recruiting. 


Conclusions 

Rankings have received a lot of criticism which is based not only on the method adopted but also on the 
philosophical approach. University rankings have changed the way higher education institutions are perceived, 
presented, and assessed and their need is not disputed. The methodology developed herein focuses on research 
as a basis for comparison using data published in international databases by extending previous work of the 
authors. The present paper examines an indicator that has hardly been investigated and that of the CHC of 
departments. It demonstrates how the different positions of academic members of staff can contribute to the 
research evaluation of their institute at different time periods. The methodology proposed takes into 
consideration the contribution in terms of mean number of P and C listed in EconLit for the different positions 
of academic members of staff. It then produces an overall performance Composite Index which is the weighted 
mean of P and L for a specific academic position expressed as a percentage of the total number of academic 
staff at different time periods and finally ranks institutes. 
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The research examined initially the scenario where the CHC of a department in subsequent periods 
consists of members who were in the department at period SR only and subsequently the case where the CHC 
does not change through time. Clearly, there are many more combinations that can be investigated. The results 
of the case studies demonstrated in the first scenario that the AUEB is constantly first in rankings with the 
exception of the last period ( LR3 ). Some universities improve, although at low position at present (SR), and 
others do not manage to maintain their position in time, and drop in ranking. In the case of the second scenario, 
the results are quite conflicting since the universities that were low on the first case, such as ARIS and UPIR 
are now on the top ranks. It is clear therefore that the assumptions of the CHC is a very sensitive factor and can 
dramatically influence the overall ranking of institutes. 

Although the methodology presented can be further enriched including data from other databases used 
more recently, such as Scopus, it certainly offers a basis for discussion and evaluation. It provides to institutes a 
tool, besides the actual ranking, for benchmarking, monitoring, and future improvement. Ultimately, such a 
methodology could help decision-makers to focus on strengths and roles within the national system and perhaps 
more than that, even at a European level, establishing a policy framework for development. 

In the Greek context, although the principle of evaluation is not yet common practice in Greece, it is 
evident that as it happens elsewhere, evaluations of academic members will become a standardised practice in 
the near future. Considering that (see section “A Conceptual Model for Ranking Universities”), with the recent 
law change in September 2011, which alters the positions that can have a tenure or tenure track from 4 to 3, the 
research can move a step further to re-evaluate the academic map and observe where this change modifies the 
conclusions produced in the present paper. 
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